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Abstract 

This paper considers the design of assessment for students of design 
according to behaviourist versus experiential pedagogical approaches, 
relating these to output-oriented as opposed to process-oriented assessment 
methods. It is part case study and part recognition of the importance of 
process in design education and how this might be applied in other 
disciplines generally, through the use of visual thinking and assessment. 
Making use of experience gained from The Open University’s entry’ level 
design course, U101: Design Thinking, the main assessment software 
(CompendiumDS) is described and presented as an alternative to 
‘convergent endpoint ’ artefacts of assessment. It is argued that the software 
and assessment design allow the evaluation of ‘unseen thinking’, providing 
an immediate focus on process rather than deterministic or behaviourist 
outcomes alone. Moreover, this evaluation can be applied at scale, without 
extensive changes to existing systems and may offer a compromise between 
measuring outcomes and the value of student-centred learning experiences. 
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Introduction: Education is not Learning Outcomes 

Tests or inquiry? 

A 2013 special edition of the Journal of Learning Design (Franz, Osborne, & Lloyd, 2013) 
referred to a challenging article by Colomina, Choi, Galan, and Meister (2012), in which the 
continued growth of professionalisation in educational institutions was presented as (at least) a 
challenge to innovative development of pedagogies, curricula and learning methods. This tension 
between measuring and predicting what we can achieve in education and innovating or responding 
to changing contexts and challenges is one familiar to anyone working in higher education today 
and has been raised in number of educational contexts. 

At policy level, Gleeson and Donnabhain (2009) observed the growing model of bureaucratic 
accountability in Irish education, suggesting that certain countries are becoming preoccupied with 
performativity as opposed to focusing on the students’ experience and process. Robinson (2010) 
observed that modern systems and curricula of education are still based on principles of ‘economic 
utility’ with change in education simply an improvement on existing models and paradigms, often 
at the expense of individual learner’s desires or motivations. Chomsky (2012) noted that “the other 
approach to education is that of Indoctrination ” and that many current systems offer 
indoctrination, effectively trapping students into such a system by treating education as a market 
and students, by default, as the necessary customers. 
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At the operational level. Bloom’s taxonomy (Bloom, 1984) still stands as one of the cornerstones 
of educational curriculum design, despite criticisms of its essentially behaviourist agenda and the 
linearity of hierarchy (see, for example, Furst, 198, Ormell, 1974). Anderson and Krathwol’s 
(2001) update of Bloom’s Taxonomy certainly helps address these concerns by introducing subtly 
(but very significantly) different cognitive elements. One notable difference between these two 
taxonomies is particularly relevant to this paper - the first was concerned with “things” (such as 
Knowledge and Comprehension) while the second introduced “actions” (such as Remembering 
and Understanding). 

This difference is at the heart of this paper - the difference between a deterministic and 
extrinsically-driven view of education, contrasted with an emergent and intrinsic (or learner- 
centred) vision. Returning to Chomsky(2012) to sum up the dilemma for educators: “do you train 
for passing tests, or do you train for creative inquiry?” 


Tick boxes or ideas? 

This is also a challenge in design education and a key starting point for this paper. On the one hand 
we hope to develop students capable of genuinely creative thinking and ability - activity that goes 
beyond the simple behaviours that might be associated with a profession or discipline. But on the 
other hand, we require educational institutions to somehow measure and demonstrate the delivery 
of such graduates - whilst doing so with fewer resources. 

In design education, it is easy to think that we are in some way immune to the dangers of the 
culture of the “tick box”, perhaps believing that the subjective of nature of design ensures that it is 
not susceptible to objective measures or criteria of assessment. But it could also be argued that 
design education is one of the most deterministic forms of pedagogy because it requires 
indoctrination into a way of behaving and thinking in itself - even more so if the design subject is 
discipline focused. If the measure of successful indoctrination is by the subjective judgement of 
those already in the discipline, then this is just as much a “tick box,” albeit one with criteria that 
are at least partially hidden. 

Increasing calls for creativity and innovation in graduates (Craft, 2006) require the development of 
a different set of behaviours - indeed, many of these are not even behaviours. It is suggested that 
by considering alternative methods and modes of design education we can begin to experiment and 
implement genuine alternatives to existing paradigms. This position arises partly from the 
epistemology of design in itself - that we can accrue new knowledge by engaging in the act of 
design. But it is also proposed because it has been demonstrated as being effective before. For 
example, the previously cited Colomina et al. (2012) presents a compelling history of this in 
design education and it is something observed in the case study presented in this paper. 

Tensions or difficulties? 

This paper is particularly concerned with the tension between the behaviourist and experiential 
approaches to design education and how mediation of these might be achieved. In particular, the 
paper responds to two further triggers. A first particular aim is to address Gleeson and 
Donnabhain’s (2009) call for alternative indicators that will “achieve balance between process and 
product, between responsive and contractual accountability and between individual and system 
outcomes” (p. 27). It is intended to show that the method and system of assessment presented also 
mediates between behaviourist and experiential approaches introduced earlier. 

Secondly, this paper aims to consider Lawson’s (1980) observation that taking an experiential 
approach in design education is particularly difficult because “there is not a lot of action to be seen 
[in observing the designer in action] and what is there cannot be readily understood” (p. 216). 
Further, it is intended to show that the learning design, assessment methods and technologies 
presented in this paper go some way to actually seeing and understanding the thinking of design 
students studying at a distance. 
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Case study or position paper? 

This paper is part case study in that we are examining the design and implementation of a system 
of learning, teaching and assessment in design education. It is a reflective piece exploring a single 
major aspect of the open online distance learning course U101: Design Thinking. It therefore does 
not quite fulfd the complete requirements advocated by Boling (2010) or Howard (2011), 
principally because it extends the question proposed by both authors as the essence of a case study, 
“How did the design come to be as it is?” This paper considers the design as embodied in the 
artefact in practice (design as a noun and verb) and, more importantly, that this process continues 
into operation and practice. 

As with the practice of design itself, design education inhabits the middle ground between 
objective and subjective epistemologies, dealing with “the ‘halfway’ between people and things” 
(Koskinen et al., 2011, p. 204) or the multiple readings of reality (Charmaz, 2000). This 
necessarily requires design education to become as adaptive as the discipline(s) it supports and, as 
with the practice of design, this paper presents the idea that we take grounded approaches whilst at 
the same time maintaining ideals and values that we deem important. Unlike any other discipline, 
design requires both the theoretical idea and the tangible output, the essence of Cross’s notion of 
designerly ways of knowing (Cross, 1982). It seems natural, therefore, that the teaching and 
learning artefacts should be no less emergent and adaptive. 

The case study: Reading minds 

The context 

The Open University is the largest provider of distance-learning higher education in the UK, 
allowing students to study at under- and post-graduate level on a part time basis, without having to 
physically attend an institution. Students study individual courses (or modules) that contribute to 
their overall study plan and these are provided as physical or online material using an online 
Virtual Learning Environment (VLE). This material is specifically designed and written to allow 
learning at a distance, containing everything students require in terms of information, activities and 
assessment materials. 

Students are also supported by a regional tutor responsible for around 20 students in a single tutor 
group. Tutors are the direct point of contact for students studying at a distance and provide support 
in the subject area, general study skills and advice and pastoral care. Tutors on the course 
presented in this paper have been largely selected from design practice backgrounds adding a 
further positive dimension to students’ experience (Lloyd, 201; Saddler 1989). The tutor group and 
tutor-student relationship is an essential component of the OU model of learning and perhaps the 
most significant differentiator to other distance learning institutions. 

A further key aspect of The Open University is its open access policy - students require no 
previous qualifications or evidence of study to undertake a course. This ensures that the university 
maintains a very diverse student population in all demographic senses. In design education, it has a 
particular significance in terms of “self-selection,” where other institutions will find themselves 
with students who have some specific notion of a particular discipline-oriented career or study 
path (Lloyd & Jones, 2013). Put together, this model of distance education is the Supported Open 
Learning (SOL) model (Ison, 2000). 

Design education at the Open University has been provided since the 1970s and the unique nature 
of the institution has required that it always took a general rather than specialist view of design and 
design education (Holden, 2009). This arguably allowed the unique formulation of the early 
notions of design thinking through the work of the Open University’s Design Group. Design at 
the OU is necessarily focused on the individual student’s methods rather than the domain specific 
output exhibited in discipline focused courses. 

In the newest course (at time of writing), U101: Design Thinking , the focus is entirely on design 
thinking and doing - in other words the process of design. The course requires students to learn 
design thinking by doing. Students are challenged with activity from the very start of the course 
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and these are continually encountered in the course material as students’ progress. A variety of 
learning material is provided to allow students to make use of different modes of learning, 
permitting genuine blended learning to emerge. In fact, it is thought that the design and conception 
of the module itself taps into deeper emotional and metaphorical connections with students ( Jones 
& Lloyd, 2013). 

A typical Open University course will take years to design and develop, with considerable 
resources going into the production of material and the design of the learning and assessment 
elements. U101: Design Thinking began its design life in 2008 and had its first live student 
presentation in 2010. To date (January 2014), 1,805 students have successfully completed the 
course. 


The artefact 

The course is divided into four blocks of six weeks study, each of which have a substantial “scale 
theme” associated with all the learning material and activities (ranging from individual design 
thinking to global contexts). At the end of each block, students submit a piece of work online that 
is formally assessed by their tutor who also provides written tuition feedback. 

This continuous, discrete assessment is a key feature of the OU assessment system and provides 
students with summative and formative checkpoints to reflect on their progress and to develop 
their work. The formal mechanism for assessment satisfies the academic standards required by the 
institution and the detailed tuition feedback method provides critical advice and feedback on 
students’ progress and work. This is known as the Tutor Marked Assessment (TMA). 

For U101: Design Thinking, this method of tuition and assessment is absolutely critical in 
allowing student designers to learn a process of design thinking for themselves. A key feature is 
the focus on the assessment of process, not final product, in a medium that fits within existing 
systems and procedures. In order to deliver this in a distance learning environment, software was 
specifically developed for the course. This software, CompendiumDS, is a “digital whiteboard” in 
which nodes can be arranged spatially similar to a mind-map (Figure 1). Nodes can consist of a 
variety of media and these can be connected to allow patterns to be represented, or “maps.” These 
maps are created by students and communicate their design process. 


30 








Journal of Learning Design 


Jones 



Figure 1. CompendiumDS software showing the interface and node types used in U101: Design 

Thinking (screen shot) 


The CompendiumDS software was developed to be as simple as possible to use making use of 
“drag and drop” or visual interaction wherever possible. Students can quickly populate a map with 
artefacts and then experiment with setting out these maps - or create a process they might wish to 
go through by actively using CompendiumDS to “design their process” as well as communicate it. 

The entire assessment process was designed to fit into existing institutional systems, a critical 
design constraint from the start. A typical offering of U101: Design Thinking will have a student 
population of around 500 students, meaning that assessment has to be handled efficiently and 
effectively whilst still maintaining the standards required for academic and discipline rigour. The 
output from CompendiumDS is a file that stores the data (as native file formats) and map (as xml) 
in a single package, readable using the software on any major operating system. This file is then 
submitted online and processed at the institutional level. Tutors then assess and provide feedback 
to students on their work. In parallel to this, a sample of students’ work and tutors’ assessment is 
itself checked for quality and consistency of assessment and tuition feedback 

Four substantive Tutor- Marked Assessments (TMA01 - TMA04) are required in the module 
covering most of the learning objectives. Students also submit a final End of Module Assessment 
(EMA) to complete the learning outcomes and produce a portfolio of work. Each TMA is based on 
the “scale” of the block it relates to and builds on skills and processes developed in previous work. 

The premise is that, as students engage in increasing “scales” of design contexts, they also have 
greater freedom to represent the process they undertake as they work towards a concept proposal 
or embodied design. Each TMA gradually develops students’ skills in familiarising themselves 
with this mode of communication until they are confident enough to complete blank maps 
according to how they wish to communicate their process of design thinking. As they do so, they 
are required to engage in active reflection on the processes they are going through. 
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Figure 2. CompendiumDS map layout for TMA01, showing the basic design process approach 

for students’ first assessment. 


It might already be clear that the pre-populated maps in early TMAs are representations of design 
processes themselves. For example, in TMA01 (Figure 2), students were asked to undertake three 
activities in an exploration phase which then led to three idea developments in the next stage. It is 
very rare that students are unable to understand and make use of this simple process of “seeing and 
doing” and quite often the results (Figure 3) exceed anything that might usually be expected from 
such a simple process (Lloyd & Jones, 2013). 



Figure 3. Some example design outputs from TMA01. 


For this paper, a single example is provided to give a sense of the outcomes that can be achieved. 
It will focus on the second assessment, TMA02, which is broken up into 4 weeks of activity, each 
week represented using nodes in the TMA map (Figure 4). 
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Figure 4. Overall CompendiumDS map for TMA02. 


Each week contains a “sub-map” node which allows further nodes to be added inside it to 
communicate sufficient detail of the substantive activity at each stage of the process (Figure 5). As 
students progress through the weeks, they work through an iterative design process by recording 
their activity and thoughts. At each stage (each week) students are also required to reflect on their 
process and describe particular formative events that have occurred. 



Figure 5. CompendiumDS ‘sub-maps’ for TMA02. 


The design layout of this particular TMA has already seen several iterations since the start of the 
course, mainly to deal with the complexity of the activities required and to respond to student 
feedback. This iteration and improvement is an essential aspect of the overall process - the design 
is never “complete” and has to continue to respond to its context and users. 
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The experience 

The experience of this form of assessment must considered from two points of view: the student 
and tutor. The tutor is attempting to understand and empathise with the student’s thinking in order 
to help develop it through feedback. The motivation behind this is to “close the gap” between 
where the student is and where they need to be - a classic definition given by Ramaprasad (1983). 
For feedback to work, the student first has to understand it. 

An original curriculum design intent for the course was that, rather than describing exactly what it 
is we wish students to be able to appear to do (what behaviours we might wish to see copied), we 
considered what the student will experience (what activities we might hope they engage in or 
discover for themselves). By allowing students to demonstrate their own methods and approaches, 
tutors have no choice but to take a student-centred approach - in effect, to try to understand what 
the student was thinking and doing in going about their work. It is proposed that the format and 
structure of assessment in the course allows precisely that. 


Figure 6 shows a student’s TMA 02 as the tutor would see it. 
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Figure 6. Student A’s TMA02 submitted assessment CompendiumDS map. 


At a glance, the tutor can readily see some of the key stages in the overall process. The top row of 
nodes represent the substantial activity and content that lead to the outputs presented in the second 
row. This second row provides the tutor with an immediate overview of what was achieved. The 
software allows tutors to see further detail by rolling over images (Figure 7) and text (Figure 8) in 
the nodes, providing an opportunity to read the map quickly and effectively. 
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Figure 7. Student A’s TMA02 CompendiuniDS map showing tutor rollover to view image. 
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This solution was chosen as it was the one that started 
to be developed in the creative session. The main 
problem identified with the original box is the 
structural weaknesses in the lid, so the idea of having 
lid and box body as 1 piece would resolve the issue. 
The box itself, while a ‘problem", had some advantages. 
Easy to manufacture, easy to transport, easy to store. 
The box also serves as the puzzle instructions - a 
feature I didn’t want to lose. A softer bag would resolve 
the issue of the lost pieces, but the cardboard served 
as protection, so while we were playing with the bag 
idea, it soon turned back to being more box-like. The 
end solution may end up being a combination of more 
than one idea, with the idea of the end result being not 
too far away from the original problem, but sleeker and 
less prone to damage. 

The solution sketch resolves the storage issue, 
overcomes the structural issue of opening the box 
repeatedly creating a weakness, as well as giving 
additional functionality in the form of a board on which 
to create the puzzle - an accidental useful side-effect. 
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Figure 8. Student A’s TMA02 CompendiuniDS map showing tutor rollover to view text. 


The simple expediency of being able to preview images and text quickly cannot be overlooked. 
Tutors wish to quickly understand what the student is thinking and doing at each part of the 
process being presented. In doing so, they are recreating (in some sense) the process the student 
has undertaken. For example, Figure 9 shows a “sub-map” of Figure 6, providing evidence of the 
actual work that has taken place. In this example, a tutor can quickly see (both literally and 
conceptually) the depth of activity and thinking that has gone into this part of the work. 


2014 Vol. 7 No. 1 


35 





















Journal of Learning Design 


Jones 


g* ' 



START - What prunes 

Problem? 



Problem 2 - 
Spilled drinks 


*■ 


Problem 3 - 
Cables 

■ 


Problem 4 - 
Giant windows 



Problem 6 - 
Nespresso 
capsule torture 
device 



Problem 7 • On 
street parking 



END - My 
Decision 


Figure 9. Problem identification in Week 1 (cf. Figure 6) 


This view then allows tutors to provide specific feedback on the process the student has gone 
through, directing students to look at particular nodes and make observations or pose questions. 
From student feedback on tutor feedback, we are confident that comments around particular nodes 
and artefacts do allow students to recall and reflect on particular aspects of their process more 
directly. This allows tutors to reinforce students’ confidence in those activities that have been 
beneficial to the process and to reflect further on those that have not. In fact, the type of 
observations that are made by tutors can be quite subtle and nuanced, going way beyond what 
might be expected from the assessment artefact itself. 

A further critical aspect of all assessments are the reflection nodes (see bottom row of nodes in 
Figure 4) where students are required to discuss their thoughts on the process they have undertaken 
in each week. Students are encouraged to consider their emotional and personal responses to the 
process they go through, starting in the first TMA by considering what they like or dislike; what 
they thought went well or not so well; and (most importantly) what they would do differently if 
they were to repeat the work. By focusing on their personal (and conscious) response to the work 
we are trying to engage students in the process of reflection-in-action (Schon, 1987) by 
consciously externalising their reflection. This mechanism is essentially the same as Sadler’s 
(1989) premise that, for effective summative learning, students should monitor the quality of their 
own work during production. 

This synchronous reflection is also reinforced by asynchronous summative reflection of the 
feedback received from the tutor. Walker (2009) discussed this form of feedback as a mechanism, 
stressing the importance of the students’ understanding of the feedback to allow this to take place - 
with reference to Biggs and Tang (2011) in terms of taking a student-centred approach to allow 
this to be more effective. It is argued that the ability to view and feedback on process allows a 
more effective form of feedback to take place: that by communicating to a student using their own 
representations of process a tutor is immediately in a student-centred position. 

The form of assessment presented here allows the student and tutor to “close the gap” between the 
production stage and the artefact submitted for assessment. In many ways, assessing the final 
outcome is perhaps too late for many students and may also miss critical elements in the process. 
By assessing process and providing feedback directly on this (and doing so in a visual way) it is 
argued that students have a greater opportunity to develop their abilities during production. 


The reflection 

CompendiumDS has been in use since the start of U101: Design Thinking and in this time we have 
built up a considerable community of knowledge around assessment of the design thinking process 
using the software as well as evidence of student success and feedback from students themselves. 

The narrative presented here is typical in many ways but it is also atypical in that it represents a 
student who has actually “got it” (possibly even in the material just presented) - that is, they have 
demonstrated a level of internalisation of a number of cognitive processes, attitudes and 
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behaviours. It is suggested that they have demonstrated several of Anderson and Krathwol’s 
(2001) taxonomic levels as embodied design activity - they have not simply created, they have 
applied, analysed and evaluated as well. 

For some students (as with educators), however, the very notion of non-behaviourist learning is 
incredibly difficult to understand or believe in. The word “believe” is used here deliberately since 
the evidence from student feedback is showing that cognitive dissonance occurs for a certain 
percentage of students or, at least, a significant “resistance to learning” (Atherton, 1999). That is, 
the ideas being presented in the course are incompatible with their own worldview of education 
and/or design. For some students, only a deterministic and behaviourist approach is ever 
imaginable. 

In between these two extremes, sit the majority of students, who will take on board some aspects 
of the module but not necessarily complete the entire conversion to process-driven methods of 
design. Research is ongoing to determine how to improve the “reach” of these methods to these 
students, for example, multimedia feedback is currently being trialled to engage student-tutor 
dialogue beyond the current linear model. 


The Analysis 

In the introduction to this paper, a number of dualities and contrasts were presented. For each of 
these dualities, it is argued that the method of assessment presented demonstrates that, as with the 
discipline of design itself, it is possible to inhabit a middle ground of ideologies and approaches. 

Firstly, it is entirely possible to create learning material and assessment artefacts that embody both 
deterministic and subjective elements. It is hoped that this is clear from the case study presented - 
it is both a deterministic and student-centred method of assessment. This is possible because it is 
the process that is assessed, not the output, meaning that the entire assessment system is 
necessarily student-centred from the start. The change to existing paradigms is simply this: 
experiential processes are assessed using objective frameworks (not behaviourist outcomes), 
within which the individual student can be considered using experiential criteria. The mediating 
factor, the tutor, is still absolutely vital in taking a “human” view of student work but this is still 
perfectly possible within a deterministic assessment framework. This observation is certainly not 
new or unique. What is important here is the context in which this takes place: distance education 
at scale. It is argued that this presents an alternative in line with Gleeson and Donnabhain’s (2009) 
call for alternative indicators. 

Secondly, it is also possible to develop behaviours at the same time as taking a student-centred 
approach to teaching and learning. One of the interesting features of the summative aspect of the 
assessment presented is that it starts with a behaviour (creating something) and then actually 
works backwards to embody other cognitive elements such as analysis or reflection. Returning to 
Anderson and Krathwol’s (2001) updated taxonomy, it is argued that the hierarchy presented is not 
necessarily a progressive model. That is, it is not simply the case that students will begin with 
Remembering, moving on to Understanding, and so on. In fact, students may start at any level or 
may even work across several. It is argued that the form of assessment described in this paper 
allows this to take place - for example, students may create without knowing how or why (and that 
then becomes their next learning goal through summative feedback). The key to this aspect is to 
realise that any taxonomy is an ideal - that, in the real world of students and tutors, individual 
approaches matter. Having systems and processes that allow this to be recognised and developed 
is vital. 

Finally, it is argued that by students presenting their work in this way it is actually possible to 
“see” what they are thinking, answering Lawson’s (1980) dilemma of not being able to read 
designers’ minds. It is not proposed that this is in any way a literal glimpse into the thinking of 
students - at this stage in the research it can only be described as, at least, a view of what the 
student “thinks they are thinking.” But even this is still a significant step beyond the further 
remove of starting with the final artefact with no reference to the process at all. 
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Summary 

In summary, it is contended that U101: Design Thinking demonstrates that accepted paradigms of 
existing modes of design education are not the only approach that can be taken in design 
education. Moreover, the positive effect on both student outcomes and experience is such that 
these alternative approaches are well worth exploring further, whether or not the context is one of 
distance learning. The success of this simple change to communication and assessment of process 
using visual mapping software allows students and tutors to engage in a significantly valuable 
dialogue. By supporting this method with institutional frameworks and standard practices, the 
tensions between “tick boxes and ideas” in learning can be resolved; the behaviours can be 
considered as part of an experiential process; and all this can be accomplished in a systematic way 
that allows large numbers of students to be considered as individual learners. 

This comes from and is supported by taking a design thinking approach to learning itself. By 
treating the learning design as a design process itself, we are able to avoid polarised positions - 
they can become embodied in the process itself. And this process continues. U101: Design 
Thinking is constantly evaluated, changed and exposed to critical review from academics, tutors 
and students. It is perhaps this responsiveness that matters most rather than which polarised 
position is taken. 
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